Conversation
github-actions
Bot
added
sqllogictest
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25602 +/- ##
==========================================
- Coverage 82.50% 82.50% -0.01%
==========================================
Files 1140 1142 +2
Lines 438050 438450 +400
Branches 438050 438450 +400
==========================================
+ Hits 361410 361730 +320
- Misses 54833 54880 +47
- Partials 21807 21840 +33 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
sunchao
reviewed
Sep 22, 2026
sunchao
left a comment
sunchao
left a comment
Member
There was a problem hiding this comment.
Thanks for working on this. I compared adbb3ce with base 714956b and found two regressions, detailed inline. Both have executable base/head reproductions. The 109 pruning unit tests and 64 ordinary SQL cases per revision passed; the separate nullable-CASE and memory-limit reproducers below fail only on head.
Comment on lines
+1875
to
+1880
| // A missing `ELSE` yields NULL, which never matches, so it adds nothing. | ||
| return case | ||
| .when_then_expr() | ||
| .iter() | ||
| .map(|(_, then)| then) | ||
| .chain(case.else_expr()) |
Member
There was a problem hiding this comment.
[P1] Preserve the implicit NULL branch during full-match inference
Omitting the missing ELSE is conservative for ordinary filtering, but this rewriter is also used to build the inverse predicate that proves an entire Parquet row group matches. That makes this change return incorrect rows when datafusion.execution.parquet.pushdown_filters=true.
I reproduced this with required Int64 columns (a,b), two one-row groups containing (2,1) and (1,2), and:
SELECT * FROM t
WHERE NOT (CASE WHEN a = 1 THEN false END) AND b > 0
ORDER BY a;Base 714956b3 returns only (1,2); head adbb3ce4 also returns (2,1), although its predicate result is NULL. With LIMIT 1 instead of ORDER BY, head returns the invalid (2,1). Disabling filter pushdown restores the correct result.
The forward rewrite falls back for NOT CASE, so it still permits full-match inversion. The inverse exposes CASE ... OR b <= 0; dropping the implicit NULL branch lets it prune both groups, which are then marked fully matched and skip row filtering. Head reports two fully matched groups and row_filter_skipped_fully_matched=1.
Could we keep CASE expressions without an explicit ELSE conservatively unhandled here, and add an end-to-end regression test? The join-generated CASE already has an explicit ELSE. Merely setting has_filter_semantics_only inside this branch would miss the forward NOT CASE path, which never descends into it.
Contributor
Author
There was a problem hiding this comment.
Good point, added a bail-out for that case as well as an SLT.
Comment on lines
+3113
to
+3114
| let pruning_bitmap = match (left_values.as_slice(), bounds.as_ref()) { | ||
| ([keys], Some(bounds)) if !keys.is_empty() => bounds |
Member
There was a problem hiding this comment.
[P2] Skip bitmap construction when dynamic filtering is inactive
bounds can exist solely for perfect-hash-join candidacy even when should_compute_dynamic_filters is false. This branch still constructs and reserves a pruning bitmap before those bounds are cleared below, so a join with dynamic filtering disabled can now fail for memory that provides no pruning benefit.
Using a native CollectLeft join with 151 Int64 build keys i * 10_000 (i = 0..151), probe keys [0, 500_000, 1_500_000], and enable_join_dynamic_filter_pushdown=false (otherwise default configuration), base 714956b3 succeeds in a 100,000-byte memory pool with 6,220 bytes reserved. Head adbb3ce4 fails with ResourcesExhausted requesting another 131,072 bytes. At a 1,000,000-byte limit both return the exact expected rows, but head reserves 137,292 bytes. I also reproduced the same failure with a Full join.
Could we gate this bitmap construction on should_compute_dynamic_filters and cover the disabled-filter case with a memory-limit regression test?
Contributor
Author
There was a problem hiding this comment.
Nice catch. I also noticed that the same problem happens for the InList path too.
So instead of guarding just the (bit)map path with should_compute_dynamic_filters, i made it guard both by extending the PushdownStrategy::Empty arm with || !should_compute_dynamic_filters; let me know if you see issues with this.
Also a unit test added.
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 23, 2026 07:20
6c48fab to
a6d5391
Compare
jayzhan211
reviewed
Sep 23, 2026
Contributor
|
run benchmarks |
Contributor
|
I think we need benchmark number |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 24, 2026 05:30
90415e6 to
ed6ba05
Compare
Contributor
Author
Looks like there's a genuine improvement for TPCH Q18, but it's a side-effect of the changes not really benefiting from the bitmap pruning in that case. That side-effect is wiring the |
Contributor
|
Per file, the adapter often rewrites the probe child to - let column = column_expr.downcast_ref::<phys_expr::Column>()?;
+ let column = match column_expr.downcast_ref::<phys_expr::CastExpr>() {
+ Some(cast) => {
+ let column = cast.expr().downcast_ref::<phys_expr::Column>()?;
+ let from = schema.fields().get(column.index())?.data_type();
+ if !is_integer_widening(from, cast.cast_type()) {
+ return None;
+ }
+ column
+ }
+ None => column_expr.downcast_ref::<phys_expr::Column>()?,
+ };/// Casts that keep the bitmap's `u64` key ordering (sign/zero extension).
fn is_integer_widening(from: &DataType, to: &DataType) -> bool {
use DataType::*;
from == to
|| matches!(
(from, to),
(Int8, Int16 | Int32 | Int64)
| (Int16, Int32 | Int64)
| (Int32, Int64)
| (UInt8, Int16 | Int32 | Int64 | UInt16 | UInt32 | UInt64)
| (UInt16, Int32 | Int64 | UInt32 | UInt64)
| (UInt32, Int64 | UInt64)
)
} |
jayzhan211
approved these changes
Sep 24, 2026
jayzhan211
left a comment
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
A non-blocking issue left
Contributor
Author
Makes sense; added conditional casting unwrapping in both |
Map the build side's keys onto a fixed-size bitmap over their range and test container min/max against it, so a scan can skip containers whose values fall in the gaps between keys. The bitmap is bounded at 128 KiB regardless of build side size, and one whose buckets are all set is discarded, so a contiguous key set costs nothing. The key range and distinct count already exist for the bounds predicate, so a dense key set is ruled out before allocating. Integer keys only; other types go unpruned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 24, 2026 14:43
7fc666c to
4ecb570
Compare
Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 24, 2026 15:01
4ecb570 to
0908a99
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Avoid scanning redundant files/row-groups/pages in the probe side of hash joins, based on the values dictated by the build side.
What changes are included in this PR?
KeyRangeBitmapimplementation, which maps the build-side values array into a finite-sized bitmap bucket, and can answer probing questions for certain rangesbuild_predicate_expressionto build the associated pruning expression fromHashTableLookupExprusing the newKeyRangeBitmapPruningExpr, which implementsPhysicalExpron top ofKeyRangeBitmapbuild_predicate_expressionso that it now pushes down pruning forCaseExprs, since that also unlocks the partitioned hash-join scenario this pr targetsWhat is the testing strategy for this PR?
A number of unit tests added, and one SLT added.
Also verified on the query shape that motivated the original issue
Are there any user-facing changes?